{
    "cells": [
        {
            "attachments": {},
            "cell_type": "markdown",
            "id": "47de02e1",
            "metadata": {},
            "source": [
                "# Running Tune experiments with AxSearch\n",
                "\n",
                "<a id=\"try-anyscale-quickstart-ray-tune-ax_example\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=ray-tune-ax_example\">\n",
                "    <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
                "</a>\n",
                "<br></br>\n",
                "\n",
                "In this tutorial we introduce Ax, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with Ax and, as a result, allow you to seamlessly scale up a Ax optimization process - without sacrificing performance.\n",
                "\n",
                "Ax is a platform for optimizing any kind of experiment, including machine learning experiments, A/B tests, and simulations. Ax can optimize discrete configurations (e.g., variants of an A/B test) using multi-armed bandit optimization, and continuous/ordered configurations (e.g. float/int parameters) using Bayesian optimization. Results of A/B tests and simulations with reinforcement learning agents often exhibit high amounts of noise. Ax supports state-of-the-art algorithms which work better than traditional Bayesian optimization in high-noise settings. Ax also supports multi-objective and constrained optimization which are common to real-world problems (e.g. improving load time without increasing data use). Ax belongs to the domain of  \"derivative-free\" and \"black-box\" optimization.\n",
                "\n",
                "In this example we minimize a simple objective to briefly demonstrate the usage of AxSearch with Ray Tune via `AxSearch`. It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume `ax-platform==0.2.4` library is installed withe python version >= 3.7. To learn more, please refer to the [Ax website](https://ax.dev/)."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 1,
            "id": "297d8b18",
            "metadata": {
                "tags": [
                    "remove-cell"
                ]
            },
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Requirement already satisfied: ax-platform==0.2.4 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (0.2.4)\n",
                        "Requirement already satisfied: botorch==0.6.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (0.6.2)\n",
                        "Requirement already satisfied: jinja2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (3.0.3)\n",
                        "Requirement already satisfied: pandas in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (1.3.5)\n",
                        "Requirement already satisfied: scipy in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (1.4.1)\n",
                        "Requirement already satisfied: plotly in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (5.6.0)\n",
                        "Requirement already satisfied: scikit-learn in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (0.24.2)\n",
                        "Requirement already satisfied: typeguard in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (2.13.3)\n",
                        "Requirement already satisfied: gpytorch>=1.6 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from botorch==0.6.2->ax-platform==0.2.4) (1.6.0)\n",
                        "Requirement already satisfied: torch>=1.9 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from botorch==0.6.2->ax-platform==0.2.4) (1.9.0)\n",
                        "Requirement already satisfied: multipledispatch in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from botorch==0.6.2->ax-platform==0.2.4) (0.6.0)\n",
                        "Requirement already satisfied: MarkupSafe>=2.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from jinja2->ax-platform==0.2.4) (2.0.1)\n",
                        "Requirement already satisfied: pytz>=2017.3 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from pandas->ax-platform==0.2.4) (2022.1)\n",
                        "Requirement already satisfied: numpy>=1.17.3 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from pandas->ax-platform==0.2.4) (1.21.6)\n",
                        "Requirement already satisfied: python-dateutil>=2.7.3 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from pandas->ax-platform==0.2.4) (2.8.2)\n",
                        "Requirement already satisfied: tenacity>=6.2.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from plotly->ax-platform==0.2.4) (8.0.1)\n",
                        "Requirement already satisfied: six in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from plotly->ax-platform==0.2.4) (1.16.0)\n",
                        "Requirement already satisfied: joblib>=0.11 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-learn->ax-platform==0.2.4) (1.1.0)\n",
                        "Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-learn->ax-platform==0.2.4) (3.0.0)\n",
                        "Requirement already satisfied: typing-extensions in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from torch>=1.9->botorch==0.6.2->ax-platform==0.2.4) (4.1.1)\n",
                        "\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\n",
                        "\u001b[0m"
                    ]
                }
            ],
            "source": [
                "# !pip install ray[tune]\n",
                "!pip install ax-platform==0.2.4"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "id": "59b1e0d1",
            "metadata": {},
            "source": [
                "Click below to see all the imports we need for this example."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 2,
            "id": "cbae6dbe",
            "metadata": {
                "tags": [
                    "hide-input"
                ]
            },
            "outputs": [],
            "source": [
                "import numpy as np\n",
                "import time\n",
                "\n",
                "import ray\n",
                "from ray import tune\n",
                "from ray.tune.search.ax import AxSearch"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "id": "7b2b6af7",
            "metadata": {},
            "source": [
                "Let's start by defining a classic benchmark for global optimization.\n",
                "The form here is explicit for demonstration, yet it is typically a black-box.\n",
                "We artificially sleep for a bit (`0.02` seconds) to simulate a long-running ML experiment.\n",
                "This setup assumes that we're running multiple `step`s of an experiment and try to tune 6-dimensions of the `x` hyperparameter."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 3,
            "id": "0f7fbe0f",
            "metadata": {},
            "outputs": [],
            "source": [
                "def landscape(x):\n",
                "    \"\"\"\n",
                "    Hartmann 6D function containing 6 local minima.\n",
                "    It is a classic benchmark for developing global optimization algorithms.\n",
                "    \"\"\"\n",
                "    alpha = np.array([1.0, 1.2, 3.0, 3.2])\n",
                "    A = np.array(\n",
                "        [\n",
                "            [10, 3, 17, 3.5, 1.7, 8],\n",
                "            [0.05, 10, 17, 0.1, 8, 14],\n",
                "            [3, 3.5, 1.7, 10, 17, 8],\n",
                "            [17, 8, 0.05, 10, 0.1, 14],\n",
                "        ]\n",
                "    )\n",
                "    P = 10 ** (-4) * np.array(\n",
                "        [\n",
                "            [1312, 1696, 5569, 124, 8283, 5886],\n",
                "            [2329, 4135, 8307, 3736, 1004, 9991],\n",
                "            [2348, 1451, 3522, 2883, 3047, 6650],\n",
                "            [4047, 8828, 8732, 5743, 1091, 381],\n",
                "        ]\n",
                "    )\n",
                "    y = 0.0\n",
                "    for j, alpha_j in enumerate(alpha):\n",
                "        t = 0\n",
                "        for k in range(6):\n",
                "            t += A[j, k] * ((x[k] - P[j, k]) ** 2)\n",
                "        y -= alpha_j * np.exp(-t)\n",
                "    return y"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "id": "0b1ae9df",
            "metadata": {},
            "source": [
                "Next, our `objective` function takes a Tune `config`, evaluates the `landscape` of our experiment in a training loop,\n",
                "and uses `tune.report` to report the `landscape` back to Tune."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 4,
            "id": "8c3f252e",
            "metadata": {},
            "outputs": [],
            "source": [
                "def objective(config):\n",
                "    for i in range(config[\"iterations\"]):\n",
                "        x = np.array([config.get(\"x{}\".format(i + 1)) for i in range(6)])\n",
                "        tune.report(\n",
                "            {\"timesteps_total\": i, \"landscape\": landscape(x), \"l2norm\": np.sqrt((x ** 2).sum())}\n",
                "        )\n",
                "        time.sleep(0.02)"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "id": "d9982d95",
            "metadata": {},
            "source": [
                "Next we define a search space. The critical assumption is that the optimal hyperparameters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 5,
            "id": "30f75f5a",
            "metadata": {},
            "outputs": [],
            "source": [
                "search_space = {\n",
                "    \"iterations\":100,\n",
                "    \"x1\": tune.uniform(0.0, 1.0),\n",
                "    \"x2\": tune.uniform(0.0, 1.0),\n",
                "    \"x3\": tune.uniform(0.0, 1.0),\n",
                "    \"x4\": tune.uniform(0.0, 1.0),\n",
                "    \"x5\": tune.uniform(0.0, 1.0),\n",
                "    \"x6\": tune.uniform(0.0, 1.0)\n",
                "}"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "id": "106d8578",
            "metadata": {
                "tags": [
                    "remove-cell"
                ]
            },
            "outputs": [],
            "source": [
                "ray.init(configure_logging=False)"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "id": "932f74e6",
            "metadata": {},
            "source": [
                "Now we define the search algorithm from `AxSearch`. If you want to constrain your parameters or even the space of outcomes, that can be easily done by passing the argumentsas below."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 7,
            "id": "34dd5c95",
            "metadata": {},
            "outputs": [],
            "source": [
                "algo = AxSearch(\n",
                "    parameter_constraints=[\"x1 + x2 <= 2.0\"],\n",
                "    outcome_constraints=[\"l2norm <= 1.25\"],\n",
                ")"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "id": "f6d18a99",
            "metadata": {},
            "source": [
                "We also use `ConcurrencyLimiter` to constrain to 4 concurrent trials. "
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 8,
            "id": "dcd905ef",
            "metadata": {},
            "outputs": [],
            "source": [
                "algo = tune.search.ConcurrencyLimiter(algo, max_concurrent=4)"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "id": "10fd5427",
            "metadata": {},
            "source": [
                "The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to `1000` samples.\n",
                "You can decrease this if it takes too long on your machine, or you can set a time limit easily through `stop` argument in the `RunConfig()` as we will show here."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 9,
            "id": "c53349a5",
            "metadata": {},
            "outputs": [],
            "source": [
                "num_samples = 100\n",
                "stop_timesteps = 200"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 10,
            "id": "6c661045",
            "metadata": {
                "tags": [
                    "remove-cell"
                ]
            },
            "outputs": [],
            "source": [
                "# Reducing samples for smoke tests\n",
                "num_samples = 10"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "id": "91076c5a",
            "metadata": {},
            "source": [
                "Finally, we run the experiment to find the global minimum of the provided landscape (which contains 5 false minima). The argument to metric, `\"landscape\"`, is provided via the `objective` function's `tune.report`. The experiment `\"min\"`imizes the \"mean_loss\" of the `landscape` by searching within `search_space` via `algo`, `num_samples` times or when `\"timesteps_total\": stop_timesteps`. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute `tuner.fit()`."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 11,
            "id": "2f519d63",
            "metadata": {},
            "outputs": [
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "[INFO 07-22 15:04:18] ax.service.ax_client: Starting optimization with verbose logging. To disable logging, set the `verbose_logging` argument to `False`. Note that float values in the logs are rounded to 6 decimal points.\n",
                        "[INFO 07-22 15:04:18] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[FixedParameter(name='iterations', parameter_type=INT, value=100), RangeParameter(name='x1', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x2', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x3', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x4', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x5', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x6', parameter_type=FLOAT, range=[0.0, 1.0])], parameter_constraints=[ParameterConstraint(1.0*x1 + 1.0*x2 <= 2.0)]).\n",
                        "[INFO 07-22 15:04:18] ax.modelbridge.dispatch_utils: Using Bayesian optimization since there are more ordered parameters than there are categories for the unordered categorical parameters.\n",
                        "[INFO 07-22 15:04:18] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 12 trials, GPEI for subsequent trials]). Iterations after 12 will take longer to generate due to  model-fitting.\n",
                        "Detected sequential enforcement. Be sure to use a ConcurrencyLimiter.\n"
                    ]
                },
                {
                    "data": {
                        "text/html": [
                            "== Status ==<br>Current time: 2022-07-22 15:04:35 (running for 00:00:16.56)<br>Memory usage on this node: 9.9/16.0 GiB<br>Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.13 GiB heap, 0.0/2.0 GiB objects<br>Current best trial: 34b7abda with landscape=-1.6624439263544026 and parameters={'iterations': 100, 'x1': 0.26526361983269453, 'x2': 0.9248840995132923, 'x3': 0.15171580761671066, 'x4': 0.43602637108415365, 'x5': 0.8573104059323668, 'x6': 0.08981018699705601}<br>Result logdir: /Users/kai/ray_results/ax<br>Number of trials: 10/10 (10 TERMINATED)<br><table>\n",
                            "<thead>\n",
                            "<tr><th>Trial name        </th><th>status    </th><th>loc            </th><th style=\"text-align: right;\">  iterations</th><th style=\"text-align: right;\">       x1</th><th style=\"text-align: right;\">       x2</th><th style=\"text-align: right;\">       x3</th><th style=\"text-align: right;\">       x4</th><th style=\"text-align: right;\">       x5</th><th style=\"text-align: right;\">        x6</th><th style=\"text-align: right;\">  iter</th><th style=\"text-align: right;\">  total time (s)</th><th style=\"text-align: right;\">  ts</th><th style=\"text-align: right;\">  landscape</th><th style=\"text-align: right;\">  l2norm</th></tr>\n",
                            "</thead>\n",
                            "<tbody>\n",
                            "<tr><td>objective_2dfbe86a</td><td>TERMINATED</td><td>127.0.0.1:44721</td><td style=\"text-align: right;\">         100</td><td style=\"text-align: right;\">0.0558336</td><td style=\"text-align: right;\">0.0896192</td><td style=\"text-align: right;\">0.958956 </td><td style=\"text-align: right;\">0.234474 </td><td style=\"text-align: right;\">0.174516 </td><td style=\"text-align: right;\">0.970311  </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         2.57372</td><td style=\"text-align: right;\">  99</td><td style=\"text-align: right;\">-0.805233  </td><td style=\"text-align: right;\"> 1.39917</td></tr>\n",
                            "<tr><td>objective_2fa776c0</td><td>TERMINATED</td><td>127.0.0.1:44726</td><td style=\"text-align: right;\">         100</td><td style=\"text-align: right;\">0.744772 </td><td style=\"text-align: right;\">0.754537 </td><td style=\"text-align: right;\">0.0950125</td><td style=\"text-align: right;\">0.273877 </td><td style=\"text-align: right;\">0.0966829</td><td style=\"text-align: right;\">0.368943  </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         2.6361 </td><td style=\"text-align: right;\">  99</td><td style=\"text-align: right;\">-0.11286   </td><td style=\"text-align: right;\"> 1.16341</td></tr>\n",
                            "<tr><td>objective_2fabaa1a</td><td>TERMINATED</td><td>127.0.0.1:44727</td><td style=\"text-align: right;\">         100</td><td style=\"text-align: right;\">0.405704 </td><td style=\"text-align: right;\">0.374626 </td><td style=\"text-align: right;\">0.935628 </td><td style=\"text-align: right;\">0.222185 </td><td style=\"text-align: right;\">0.787212 </td><td style=\"text-align: right;\">0.00812439</td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         2.62393</td><td style=\"text-align: right;\">  99</td><td style=\"text-align: right;\">-0.11348   </td><td style=\"text-align: right;\"> 1.35995</td></tr>\n",
                            "<tr><td>objective_2faee7c0</td><td>TERMINATED</td><td>127.0.0.1:44728</td><td style=\"text-align: right;\">         100</td><td style=\"text-align: right;\">0.664728 </td><td style=\"text-align: right;\">0.207519 </td><td style=\"text-align: right;\">0.359514 </td><td style=\"text-align: right;\">0.704578 </td><td style=\"text-align: right;\">0.755882 </td><td style=\"text-align: right;\">0.812402  </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         2.62069</td><td style=\"text-align: right;\">  99</td><td style=\"text-align: right;\">-0.0119837 </td><td style=\"text-align: right;\"> 1.53035</td></tr>\n",
                            "<tr><td>objective_313d3d3a</td><td>TERMINATED</td><td>127.0.0.1:44747</td><td style=\"text-align: right;\">         100</td><td style=\"text-align: right;\">0.0418746</td><td style=\"text-align: right;\">0.992783 </td><td style=\"text-align: right;\">0.906027 </td><td style=\"text-align: right;\">0.594429 </td><td style=\"text-align: right;\">0.825393 </td><td style=\"text-align: right;\">0.646362  </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         3.16233</td><td style=\"text-align: right;\">  99</td><td style=\"text-align: right;\">-0.00677976</td><td style=\"text-align: right;\"> 1.80573</td></tr>\n",
                            "<tr><td>objective_32c9acd8</td><td>TERMINATED</td><td>127.0.0.1:44726</td><td style=\"text-align: right;\">         100</td><td style=\"text-align: right;\">0.126064 </td><td style=\"text-align: right;\">0.703408 </td><td style=\"text-align: right;\">0.344681 </td><td style=\"text-align: right;\">0.337363 </td><td style=\"text-align: right;\">0.401396 </td><td style=\"text-align: right;\">0.679202  </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         3.12119</td><td style=\"text-align: right;\">  99</td><td style=\"text-align: right;\">-0.904622  </td><td style=\"text-align: right;\"> 1.16864</td></tr>\n",
                            "<tr><td>objective_32cf8ca2</td><td>TERMINATED</td><td>127.0.0.1:44756</td><td style=\"text-align: right;\">         100</td><td style=\"text-align: right;\">0.0910936</td><td style=\"text-align: right;\">0.304138 </td><td style=\"text-align: right;\">0.869848 </td><td style=\"text-align: right;\">0.405435 </td><td style=\"text-align: right;\">0.567922 </td><td style=\"text-align: right;\">0.228608  </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         2.70791</td><td style=\"text-align: right;\">  99</td><td style=\"text-align: right;\">-0.146532  </td><td style=\"text-align: right;\"> 1.18178</td></tr>\n",
                            "<tr><td>objective_32d8dd20</td><td>TERMINATED</td><td>127.0.0.1:44758</td><td style=\"text-align: right;\">         100</td><td style=\"text-align: right;\">0.603178 </td><td style=\"text-align: right;\">0.409057 </td><td style=\"text-align: right;\">0.729056 </td><td style=\"text-align: right;\">0.0825984</td><td style=\"text-align: right;\">0.572948 </td><td style=\"text-align: right;\">0.508304  </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         2.64158</td><td style=\"text-align: right;\">  99</td><td style=\"text-align: right;\">-0.247223  </td><td style=\"text-align: right;\"> 1.28691</td></tr>\n",
                            "<tr><td>objective_34adf04a</td><td>TERMINATED</td><td>127.0.0.1:44768</td><td style=\"text-align: right;\">         100</td><td style=\"text-align: right;\">0.454189 </td><td style=\"text-align: right;\">0.271772 </td><td style=\"text-align: right;\">0.530871 </td><td style=\"text-align: right;\">0.991841 </td><td style=\"text-align: right;\">0.691843 </td><td style=\"text-align: right;\">0.472366  </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         2.70327</td><td style=\"text-align: right;\">  99</td><td style=\"text-align: right;\">-0.0132915 </td><td style=\"text-align: right;\"> 1.49917</td></tr>\n",
                            "<tr><td>objective_34b7abda</td><td>TERMINATED</td><td>127.0.0.1:44771</td><td style=\"text-align: right;\">         100</td><td style=\"text-align: right;\">0.265264 </td><td style=\"text-align: right;\">0.924884 </td><td style=\"text-align: right;\">0.151716 </td><td style=\"text-align: right;\">0.436026 </td><td style=\"text-align: right;\">0.85731  </td><td style=\"text-align: right;\">0.0898102 </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         2.68521</td><td style=\"text-align: right;\">  99</td><td style=\"text-align: right;\">-1.66244   </td><td style=\"text-align: right;\"> 1.37185</td></tr>\n",
                            "</tbody>\n",
                            "</table><br><br>"
                        ],
                        "text/plain": [
                            "<IPython.core.display.HTML object>"
                        ]
                    },
                    "metadata": {},
                    "output_type": "display_data"
                },
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "[INFO 07-22 15:04:19] ax.service.ax_client: Generated new trial 0 with parameters {'x1': 0.055834, 'x2': 0.089619, 'x3': 0.958956, 'x4': 0.234474, 'x5': 0.174516, 'x6': 0.970311, 'iterations': 100}.\n",
                        "[INFO 07-22 15:04:22] ax.service.ax_client: Generated new trial 1 with parameters {'x1': 0.744772, 'x2': 0.754537, 'x3': 0.095012, 'x4': 0.273877, 'x5': 0.096683, 'x6': 0.368943, 'iterations': 100}.\n",
                        "[INFO 07-22 15:04:22] ax.service.ax_client: Generated new trial 2 with parameters {'x1': 0.405704, 'x2': 0.374626, 'x3': 0.935628, 'x4': 0.222185, 'x5': 0.787212, 'x6': 0.008124, 'iterations': 100}.\n",
                        "[INFO 07-22 15:04:22] ax.service.ax_client: Generated new trial 3 with parameters {'x1': 0.664728, 'x2': 0.207519, 'x3': 0.359514, 'x4': 0.704578, 'x5': 0.755882, 'x6': 0.812402, 'iterations': 100}.\n"
                    ]
                },
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Result for objective_2dfbe86a:\n",
                        "  date: 2022-07-22_15-04-22\n",
                        "  done: false\n",
                        "  experiment_id: 4ef8a12ac94a4f4fa483ec18e347967f\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 1\n",
                        "  l2norm: 1.3991721132671366\n",
                        "  landscape: -0.8052333562869153\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44721\n",
                        "  time_since_restore: 0.00022912025451660156\n",
                        "  time_this_iter_s: 0.00022912025451660156\n",
                        "  time_total_s: 0.00022912025451660156\n",
                        "  timestamp: 1658498662\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 0\n",
                        "  training_iteration: 1\n",
                        "  trial_id: 2dfbe86a\n",
                        "  warmup_time: 0.0035619735717773438\n",
                        "  \n"
                    ]
                },
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "[INFO 07-22 15:04:24] ax.service.ax_client: Completed trial 0 with data: {'landscape': (-0.805233, None), 'l2norm': (1.399172, None)}.\n",
                        "[INFO 07-22 15:04:24] ax.service.ax_client: Generated new trial 4 with parameters {'x1': 0.041875, 'x2': 0.992783, 'x3': 0.906027, 'x4': 0.594429, 'x5': 0.825393, 'x6': 0.646362, 'iterations': 100}.\n"
                    ]
                },
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Result for objective_2faee7c0:\n",
                        "  date: 2022-07-22_15-04-24\n",
                        "  done: false\n",
                        "  experiment_id: 3699644e85ac439cb7c1a36ed0976307\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 1\n",
                        "  l2norm: 1.530347488145437\n",
                        "  landscape: -0.011983676977099367\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44728\n",
                        "  time_since_restore: 0.00022292137145996094\n",
                        "  time_this_iter_s: 0.00022292137145996094\n",
                        "  time_total_s: 0.00022292137145996094\n",
                        "  timestamp: 1658498664\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 0\n",
                        "  training_iteration: 1\n",
                        "  trial_id: 2faee7c0\n",
                        "  warmup_time: 0.0027179718017578125\n",
                        "  \n",
                        "Result for objective_2fa776c0:\n",
                        "  date: 2022-07-22_15-04-24\n",
                        "  done: false\n",
                        "  experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 1\n",
                        "  l2norm: 1.1634068454629019\n",
                        "  landscape: -0.11285961764770336\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44726\n",
                        "  time_since_restore: 0.000225067138671875\n",
                        "  time_this_iter_s: 0.000225067138671875\n",
                        "  time_total_s: 0.000225067138671875\n",
                        "  timestamp: 1658498664\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 0\n",
                        "  training_iteration: 1\n",
                        "  trial_id: 2fa776c0\n",
                        "  warmup_time: 0.0026290416717529297\n",
                        "  \n",
                        "Result for objective_2dfbe86a:\n",
                        "  date: 2022-07-22_15-04-24\n",
                        "  done: true\n",
                        "  experiment_id: 4ef8a12ac94a4f4fa483ec18e347967f\n",
                        "  experiment_tag: 1_iterations=100,x1=0.0558,x2=0.0896,x3=0.9590,x4=0.2345,x5=0.1745,x6=0.9703\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 100\n",
                        "  l2norm: 1.3991721132671366\n",
                        "  landscape: -0.8052333562869153\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44721\n",
                        "  time_since_restore: 2.573719024658203\n",
                        "  time_this_iter_s: 0.0251619815826416\n",
                        "  time_total_s: 2.573719024658203\n",
                        "  timestamp: 1658498664\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 99\n",
                        "  training_iteration: 100\n",
                        "  trial_id: 2dfbe86a\n",
                        "  warmup_time: 0.0035619735717773438\n",
                        "  \n",
                        "Result for objective_2fabaa1a:\n",
                        "  date: 2022-07-22_15-04-24\n",
                        "  done: false\n",
                        "  experiment_id: eb9287e4fe5f44c7868dc943e2642312\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 1\n",
                        "  l2norm: 1.3599537840291782\n",
                        "  landscape: -0.11348012497414121\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44727\n",
                        "  time_since_restore: 0.00022077560424804688\n",
                        "  time_this_iter_s: 0.00022077560424804688\n",
                        "  time_total_s: 0.00022077560424804688\n",
                        "  timestamp: 1658498664\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 0\n",
                        "  training_iteration: 1\n",
                        "  trial_id: 2fabaa1a\n",
                        "  warmup_time: 0.0025510787963867188\n",
                        "  \n"
                    ]
                },
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "[INFO 07-22 15:04:27] ax.service.ax_client: Completed trial 3 with data: {'landscape': (-0.011984, None), 'l2norm': (1.530347, None)}.\n",
                        "[INFO 07-22 15:04:27] ax.service.ax_client: Generated new trial 5 with parameters {'x1': 0.126064, 'x2': 0.703408, 'x3': 0.344681, 'x4': 0.337363, 'x5': 0.401396, 'x6': 0.679202, 'iterations': 100}.\n",
                        "[INFO 07-22 15:04:27] ax.service.ax_client: Completed trial 1 with data: {'landscape': (-0.11286, None), 'l2norm': (1.163407, None)}.\n",
                        "[INFO 07-22 15:04:27] ax.service.ax_client: Generated new trial 6 with parameters {'x1': 0.091094, 'x2': 0.304138, 'x3': 0.869848, 'x4': 0.405435, 'x5': 0.567922, 'x6': 0.228608, 'iterations': 100}.\n",
                        "[INFO 07-22 15:04:27] ax.service.ax_client: Completed trial 2 with data: {'landscape': (-0.11348, None), 'l2norm': (1.359954, None)}.\n",
                        "[INFO 07-22 15:04:27] ax.service.ax_client: Generated new trial 7 with parameters {'x1': 0.603178, 'x2': 0.409057, 'x3': 0.729056, 'x4': 0.082598, 'x5': 0.572948, 'x6': 0.508304, 'iterations': 100}.\n"
                    ]
                },
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Result for objective_313d3d3a:\n",
                        "  date: 2022-07-22_15-04-27\n",
                        "  done: false\n",
                        "  experiment_id: fa7afd557e154fbebe4f54d8eedb3573\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 1\n",
                        "  l2norm: 1.805729990121368\n",
                        "  landscape: -0.006779757704679272\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44747\n",
                        "  time_since_restore: 0.00021076202392578125\n",
                        "  time_this_iter_s: 0.00021076202392578125\n",
                        "  time_total_s: 0.00021076202392578125\n",
                        "  timestamp: 1658498667\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 0\n",
                        "  training_iteration: 1\n",
                        "  trial_id: 313d3d3a\n",
                        "  warmup_time: 0.0029790401458740234\n",
                        "  \n",
                        "Result for objective_2faee7c0:\n",
                        "  date: 2022-07-22_15-04-27\n",
                        "  done: true\n",
                        "  experiment_id: 3699644e85ac439cb7c1a36ed0976307\n",
                        "  experiment_tag: 4_iterations=100,x1=0.6647,x2=0.2075,x3=0.3595,x4=0.7046,x5=0.7559,x6=0.8124\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 100\n",
                        "  l2norm: 1.530347488145437\n",
                        "  landscape: -0.011983676977099367\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44728\n",
                        "  time_since_restore: 2.6206929683685303\n",
                        "  time_this_iter_s: 0.027359962463378906\n",
                        "  time_total_s: 2.6206929683685303\n",
                        "  timestamp: 1658498667\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 99\n",
                        "  training_iteration: 100\n",
                        "  trial_id: 2faee7c0\n",
                        "  warmup_time: 0.0027179718017578125\n",
                        "  \n",
                        "Result for objective_2fa776c0:\n",
                        "  date: 2022-07-22_15-04-27\n",
                        "  done: true\n",
                        "  experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7\n",
                        "  experiment_tag: 2_iterations=100,x1=0.7448,x2=0.7545,x3=0.0950,x4=0.2739,x5=0.0967,x6=0.3689\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 100\n",
                        "  l2norm: 1.1634068454629019\n",
                        "  landscape: -0.11285961764770336\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44726\n",
                        "  time_since_restore: 2.6361019611358643\n",
                        "  time_this_iter_s: 0.0264589786529541\n",
                        "  time_total_s: 2.6361019611358643\n",
                        "  timestamp: 1658498667\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 99\n",
                        "  training_iteration: 100\n",
                        "  trial_id: 2fa776c0\n",
                        "  warmup_time: 0.0026290416717529297\n",
                        "  \n",
                        "Result for objective_32c9acd8:\n",
                        "  date: 2022-07-22_15-04-27\n",
                        "  done: false\n",
                        "  experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 1\n",
                        "  l2norm: 1.1686440476629836\n",
                        "  landscape: -0.9046216637367911\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44726\n",
                        "  time_since_restore: 0.00020194053649902344\n",
                        "  time_this_iter_s: 0.00020194053649902344\n",
                        "  time_total_s: 0.00020194053649902344\n",
                        "  timestamp: 1658498667\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 0\n",
                        "  training_iteration: 1\n",
                        "  trial_id: 32c9acd8\n",
                        "  warmup_time: 0.0026290416717529297\n",
                        "  \n",
                        "Result for objective_2fabaa1a:\n",
                        "  date: 2022-07-22_15-04-27\n",
                        "  done: true\n",
                        "  experiment_id: eb9287e4fe5f44c7868dc943e2642312\n",
                        "  experiment_tag: 3_iterations=100,x1=0.4057,x2=0.3746,x3=0.9356,x4=0.2222,x5=0.7872,x6=0.0081\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 100\n",
                        "  l2norm: 1.3599537840291782\n",
                        "  landscape: -0.11348012497414121\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44727\n",
                        "  time_since_restore: 2.623929977416992\n",
                        "  time_this_iter_s: 0.032716989517211914\n",
                        "  time_total_s: 2.623929977416992\n",
                        "  timestamp: 1658498667\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 99\n",
                        "  training_iteration: 100\n",
                        "  trial_id: 2fabaa1a\n",
                        "  warmup_time: 0.0025510787963867188\n",
                        "  \n",
                        "Result for objective_32d8dd20:\n",
                        "  date: 2022-07-22_15-04-30\n",
                        "  done: false\n",
                        "  experiment_id: 171527593b0f4cbf941c0a03faaf0953\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 1\n",
                        "  l2norm: 1.2869105702896437\n",
                        "  landscape: -0.24722262157458608\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44758\n",
                        "  time_since_restore: 0.00021886825561523438\n",
                        "  time_this_iter_s: 0.00021886825561523438\n",
                        "  time_total_s: 0.00021886825561523438\n",
                        "  timestamp: 1658498670\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 0\n",
                        "  training_iteration: 1\n",
                        "  trial_id: 32d8dd20\n",
                        "  warmup_time: 0.002732992172241211\n",
                        "  \n",
                        "Result for objective_32cf8ca2:\n",
                        "  date: 2022-07-22_15-04-29\n",
                        "  done: false\n",
                        "  experiment_id: 37610500f6df493aae4e7e46bb21bf09\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 1\n",
                        "  l2norm: 1.1817810425508524\n",
                        "  landscape: -0.14653248187442922\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44756\n",
                        "  time_since_restore: 0.00025081634521484375\n",
                        "  time_this_iter_s: 0.00025081634521484375\n",
                        "  time_total_s: 0.00025081634521484375\n",
                        "  timestamp: 1658498669\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 0\n",
                        "  training_iteration: 1\n",
                        "  trial_id: 32cf8ca2\n",
                        "  warmup_time: 0.0032138824462890625\n",
                        "  \n"
                    ]
                },
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "[INFO 07-22 15:04:30] ax.service.ax_client: Completed trial 4 with data: {'landscape': (-0.00678, None), 'l2norm': (1.80573, None)}.\n",
                        "[INFO 07-22 15:04:30] ax.service.ax_client: Generated new trial 8 with parameters {'x1': 0.454189, 'x2': 0.271772, 'x3': 0.530871, 'x4': 0.991841, 'x5': 0.691843, 'x6': 0.472366, 'iterations': 100}.\n",
                        "[INFO 07-22 15:04:30] ax.service.ax_client: Completed trial 5 with data: {'landscape': (-0.904622, None), 'l2norm': (1.168644, None)}.\n",
                        "[INFO 07-22 15:04:30] ax.service.ax_client: Generated new trial 9 with parameters {'x1': 0.265264, 'x2': 0.924884, 'x3': 0.151716, 'x4': 0.436026, 'x5': 0.85731, 'x6': 0.08981, 'iterations': 100}.\n"
                    ]
                },
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Result for objective_313d3d3a:\n",
                        "  date: 2022-07-22_15-04-30\n",
                        "  done: true\n",
                        "  experiment_id: fa7afd557e154fbebe4f54d8eedb3573\n",
                        "  experiment_tag: 5_iterations=100,x1=0.0419,x2=0.9928,x3=0.9060,x4=0.5944,x5=0.8254,x6=0.6464\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 100\n",
                        "  l2norm: 1.805729990121368\n",
                        "  landscape: -0.006779757704679272\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44747\n",
                        "  time_since_restore: 3.1623308658599854\n",
                        "  time_this_iter_s: 0.02911996841430664\n",
                        "  time_total_s: 3.1623308658599854\n",
                        "  timestamp: 1658498670\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 99\n",
                        "  training_iteration: 100\n",
                        "  trial_id: 313d3d3a\n",
                        "  warmup_time: 0.0029790401458740234\n",
                        "  \n",
                        "Result for objective_32c9acd8:\n",
                        "  date: 2022-07-22_15-04-30\n",
                        "  done: true\n",
                        "  experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7\n",
                        "  experiment_tag: 6_iterations=100,x1=0.1261,x2=0.7034,x3=0.3447,x4=0.3374,x5=0.4014,x6=0.6792\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 100\n",
                        "  l2norm: 1.1686440476629836\n",
                        "  landscape: -0.9046216637367911\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44726\n",
                        "  time_since_restore: 3.1211891174316406\n",
                        "  time_this_iter_s: 0.02954697608947754\n",
                        "  time_total_s: 3.1211891174316406\n",
                        "  timestamp: 1658498670\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 99\n",
                        "  training_iteration: 100\n",
                        "  trial_id: 32c9acd8\n",
                        "  warmup_time: 0.0026290416717529297\n",
                        "  \n"
                    ]
                },
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "[INFO 07-22 15:04:32] ax.service.ax_client: Completed trial 7 with data: {'landscape': (-0.247223, None), 'l2norm': (1.286911, None)}.\n",
                        "[INFO 07-22 15:04:32] ax.service.ax_client: Completed trial 6 with data: {'landscape': (-0.146532, None), 'l2norm': (1.181781, None)}.\n"
                    ]
                },
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Result for objective_32d8dd20:\n",
                        "  date: 2022-07-22_15-04-32\n",
                        "  done: true\n",
                        "  experiment_id: 171527593b0f4cbf941c0a03faaf0953\n",
                        "  experiment_tag: 8_iterations=100,x1=0.6032,x2=0.4091,x3=0.7291,x4=0.0826,x5=0.5729,x6=0.5083\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 100\n",
                        "  l2norm: 1.2869105702896437\n",
                        "  landscape: -0.24722262157458608\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44758\n",
                        "  time_since_restore: 2.6415798664093018\n",
                        "  time_this_iter_s: 0.026781082153320312\n",
                        "  time_total_s: 2.6415798664093018\n",
                        "  timestamp: 1658498672\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 99\n",
                        "  training_iteration: 100\n",
                        "  trial_id: 32d8dd20\n",
                        "  warmup_time: 0.002732992172241211\n",
                        "  \n",
                        "Result for objective_32cf8ca2:\n",
                        "  date: 2022-07-22_15-04-32\n",
                        "  done: true\n",
                        "  experiment_id: 37610500f6df493aae4e7e46bb21bf09\n",
                        "  experiment_tag: 7_iterations=100,x1=0.0911,x2=0.3041,x3=0.8698,x4=0.4054,x5=0.5679,x6=0.2286\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 100\n",
                        "  l2norm: 1.1817810425508524\n",
                        "  landscape: -0.14653248187442922\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44756\n",
                        "  time_since_restore: 2.707913875579834\n",
                        "  time_this_iter_s: 0.027456998825073242\n",
                        "  time_total_s: 2.707913875579834\n",
                        "  timestamp: 1658498672\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 99\n",
                        "  training_iteration: 100\n",
                        "  trial_id: 32cf8ca2\n",
                        "  warmup_time: 0.0032138824462890625\n",
                        "  \n",
                        "Result for objective_34adf04a:\n",
                        "  date: 2022-07-22_15-04-33\n",
                        "  done: false\n",
                        "  experiment_id: 4f65c5b68f5c49d98fda388e37c83deb\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 1\n",
                        "  l2norm: 1.4991655675380078\n",
                        "  landscape: -0.01329150870283869\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44768\n",
                        "  time_since_restore: 0.00021600723266601562\n",
                        "  time_this_iter_s: 0.00021600723266601562\n",
                        "  time_total_s: 0.00021600723266601562\n",
                        "  timestamp: 1658498673\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 0\n",
                        "  training_iteration: 1\n",
                        "  trial_id: 34adf04a\n",
                        "  warmup_time: 0.0027239322662353516\n",
                        "  \n",
                        "Result for objective_34b7abda:\n",
                        "  date: 2022-07-22_15-04-33\n",
                        "  done: false\n",
                        "  experiment_id: f135a2c40f5644ba9d2ae096a9dd10e0\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 1\n",
                        "  l2norm: 1.3718451333547932\n",
                        "  landscape: -1.6624439263544026\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44771\n",
                        "  time_since_restore: 0.0002338886260986328\n",
                        "  time_this_iter_s: 0.0002338886260986328\n",
                        "  time_total_s: 0.0002338886260986328\n",
                        "  timestamp: 1658498673\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 0\n",
                        "  training_iteration: 1\n",
                        "  trial_id: 34b7abda\n",
                        "  warmup_time: 0.002721071243286133\n",
                        "  \n"
                    ]
                },
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "[INFO 07-22 15:04:35] ax.service.ax_client: Completed trial 8 with data: {'landscape': (-0.013292, None), 'l2norm': (1.499166, None)}.\n",
                        "[INFO 07-22 15:04:35] ax.service.ax_client: Completed trial 9 with data: {'landscape': (-1.662444, None), 'l2norm': (1.371845, None)}.\n"
                    ]
                },
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Result for objective_34adf04a:\n",
                        "  date: 2022-07-22_15-04-35\n",
                        "  done: true\n",
                        "  experiment_id: 4f65c5b68f5c49d98fda388e37c83deb\n",
                        "  experiment_tag: 9_iterations=100,x1=0.4542,x2=0.2718,x3=0.5309,x4=0.9918,x5=0.6918,x6=0.4724\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 100\n",
                        "  l2norm: 1.4991655675380078\n",
                        "  landscape: -0.01329150870283869\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44768\n",
                        "  time_since_restore: 2.7032668590545654\n",
                        "  time_this_iter_s: 0.029300928115844727\n",
                        "  time_total_s: 2.7032668590545654\n",
                        "  timestamp: 1658498675\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 99\n",
                        "  training_iteration: 100\n",
                        "  trial_id: 34adf04a\n",
                        "  warmup_time: 0.0027239322662353516\n",
                        "  \n",
                        "Result for objective_34b7abda:\n",
                        "  date: 2022-07-22_15-04-35\n",
                        "  done: true\n",
                        "  experiment_id: f135a2c40f5644ba9d2ae096a9dd10e0\n",
                        "  experiment_tag: 10_iterations=100,x1=0.2653,x2=0.9249,x3=0.1517,x4=0.4360,x5=0.8573,x6=0.0898\n",
                        "  hostname: Kais-MacBook-Pro.local\n",
                        "  iterations_since_restore: 100\n",
                        "  l2norm: 1.3718451333547932\n",
                        "  landscape: -1.6624439263544026\n",
                        "  node_ip: 127.0.0.1\n",
                        "  pid: 44771\n",
                        "  time_since_restore: 2.6852078437805176\n",
                        "  time_this_iter_s: 0.029579877853393555\n",
                        "  time_total_s: 2.6852078437805176\n",
                        "  timestamp: 1658498675\n",
                        "  timesteps_since_restore: 0\n",
                        "  timesteps_total: 99\n",
                        "  training_iteration: 100\n",
                        "  trial_id: 34b7abda\n",
                        "  warmup_time: 0.002721071243286133\n",
                        "  \n"
                    ]
                }
            ],
            "source": [
                "tuner = tune.Tuner(\n",
                "    objective,\n",
                "    tune_config=tune.TuneConfig(\n",
                "        metric=\"landscape\",\n",
                "        mode=\"min\",\n",
                "        search_alg=algo,\n",
                "        num_samples=num_samples,\n",
                "    ),\n",
                "    run_config=tune.RunConfig(\n",
                "        name=\"ax\",\n",
                "        stop={\"timesteps_total\": stop_timesteps}\n",
                "    ),\n",
                "    param_space=search_space,\n",
                ")\n",
                "results = tuner.fit()"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "id": "860b53b0",
            "metadata": {},
            "source": [
                "And now we have the hyperparameters found to minimize the mean loss."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 12,
            "id": "12906421",
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Best hyperparameters found were:  {'iterations': 100, 'x1': 0.26526361983269453, 'x2': 0.9248840995132923, 'x3': 0.15171580761671066, 'x4': 0.43602637108415365, 'x5': 0.8573104059323668, 'x6': 0.08981018699705601}\n"
                    ]
                }
            ],
            "source": [
                "print(\"Best hyperparameters found were: \", results.get_best_result().config)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 13,
            "id": "68872424",
            "metadata": {
                "tags": [
                    "remove-cell"
                ]
            },
            "outputs": [],
            "source": [
                "ray.shutdown()"
            ]
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3 (ipykernel)",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.8.13"
        },
        "orphan": true
    },
    "nbformat": 4,
    "nbformat_minor": 5
}